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Abstract 

Background: Online medical education curricula offer new tools to teach and evaluate learners. The effect on 
educational outcomes of using learner feedback to guide curricular revision for online learning is unknown. 

Methods: In this study, qualitative analysis of learner feedback gathered from an online curriculum was used to 
identify themes of learner feedback, and changes to the online curriculum in response to this feedback were 
tracked. Learner satisfaction and knowledge gains were then compared from before and after implementation of 
learner feedback 

Results: 37,755 learners from 122 internal medicine residency training programs were studied, including 9437 
postgraduate year (PGY)l residents (24.4 % of learners), 9864 PGY2 residents (25.5 %), 9653 PGY3 residents (25.0 %), 
and 6605 attending physicians (17.0 %). Qualitative analysis of learner feedback on how to improve the curriculum 
showed that learners commented most on the overall quality of the educational content, followed by specific 
comments on the content. When learner feedback was incorporated into curricular revision, learner satisfaction with 
the instructive value of the curriculum (1 =not instructive; 5 = highly instructive) increased from 3.8 to 4.1 
(p < 0.001), and knowledge gains (i.e., post test scores minus pretest scores) increased from 17.0 % to 20.2 % 
(p< 0.001). 

Conclusions: Learners give more feedback on the factual content of a curriculum than on other areas such as 
interactivity or website design. Incorporating learner feedback into curricular revision was associated with improved 
educational outcomes. Online curricula should be designed to include a mechanism for learner feedback and that 
feedback should be used for future curricular revision. 
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Background 

One of the first steps in curriculum development is a 
needs assessment of targeted learners [1,2]. Kern et al. 
state that a curriculum that does not address the needs 
of its learners risks being inefficient or ineffective [1]. 
The role of a needs assessment does not vanish once a 
curriculum has been implemented or is moved online. 
However, many online curricula have been developed 
without attention to the principles of curriculum devel- 
opment [3,4]. This is unfortunate, because the Internet 
adds the capability of efficiently performing recurrent 
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needs assessments of its learners [5]. By incorporating 
such assessment into design of online curricula, educa- 
tors can easily determine whether the needs of learners 
have been met and gauge program effectiveness [5]. This 
information then serves as the needs assessment for the 
next round of curricular revision. Many online curricula, 
however, do not include outcomes assessment and risk 
becoming out of date [4,6]. Since a curriculum's goals 
and objectives will likely evolve over time as learning 
needs change, these rounds of outcomes assessment, 
needs assessment, and curriculum revision become im- 
portant to the long-term success of a curriculum [2]. 

In addition to enhanced capabilities with outcomes as- 
sessment, the Internet offers tools that can be used when 
educating physicians. As Cook and Dupras state: "The 



© 2012 Sisson et al; licensee BioMed Central ttd. This is an Open Access article distributed under the terms of the Creative 
Commons Attribution License (httpy/creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and 
reproduction in any medium, provided the original work is properly cited. 



Sisson et al. BMC Medical Education 2012, 12:55 
http://www.biomedcentral.com/1472-6920/12/55 



Page 2 of 7 



most effective websites creatively integrate content with 
the power and flexibiUty of the Web to enhance learning 
rather than merely replicate traditional methods." [7] 
Interactivity is one tool available to Web educators that 
cannot be replicated in a textbook [5,8,9]. The Internet 
also allows for the incorporation of multimedia into con- 
tent, including audio narration, animation, and video 
clips [7,10,11]. Hyperlinks can be used to augment con- 
tent, directing learners to source material or additional 
resources [7]. The impersonality of online training can 
be minimized by allowing for communication with 
instructors or other learners [5,7,9,11,12]. 

With all that is possible with online education, much 
has been written about what online learners prefer. 
Atreja et al. showed that the best predictors of satisfac- 
tion among the web-based learners they studied were 
the appeal of the website design and the ability of the 
course to improve subject understanding [12]. They also 
found that the ability to communicate with instructors 
and other learners was associated with greater satisfac- 
tion [12]. Others have shown learner satisfaction with 
web-based education increases when an online curricu- 
lum is interactive, with a user-friendly interface that is 
easy to navigate [5]. Still others have shown that the 
quality of the online educational content is most import- 
ant to learners [13]. Many of the features made possible 
by online learning increase learner satisfaction, but at 
the expense of an increased time commitment among 
learners [10]. 

Internet-based educational programs should be 
designed and studied to determine those features that 
learners prefer, ideally incorporating this information 
into a needs assessment for curricular revision to im- 
prove educational outcomes [2,11,14,15]. Heeding that 
call, we describe here learner feedback on a widely-used 
online curriculum on ambulatory care and evaluate asso- 
ciated changes in learner satisfaction and educational 
outcomes. 

Methods 

Data generation 

Study data was generated by the Internal Medicine 
Curriculum on the Johns Hopkins Internet Learning 
Center. The Johns Hopkins Internet Learning Center is 
an educational website established in 2002 and is 
offered to internal medicine residency training pro- 
grams that subscribe for an annual fee [16]. Residents 
and faculty at subscribing programs can then access 
educational content, which is divided into topic-specific 
training modules. Training modules are structured in a 
pretest-didactics-post test format. Learners comment 
on the website and curriculum on general message 
boards contained on the Internet Learning Center web- 
site. Starting in the 2003/04 academic year, learners 



were required to rate the instructive value of the mod- 
ule on a Likert scale (1 = not instructive; 5 = highly in- 
structive). Starting in the 2005/06 academic year, 
learners were also required to give feedback on each 
training module by answering in free text the following: 
"Please tell us how to improve this module". Learners 
could enter more than one comment or even nonsens- 
ical comments. For this study, feedback data (i.e., mod- 
ule instructive value ratings and free text comments) 
were analyzed from the 2003/04 academic year through 
the 2007/08 academic year on 19 training modules that 
were part of the curriculum each of those academic 
years. For feedback comments, 1000 comments were 
randomly selected for each of academic years 2005/06, 
2006/07, and 2007/08. Likert ratings on instructive 
value from all learners who completed any one of the 
19 training modules were included for study. The study 
was approved by the Johns Hopkins Institutional Re- 
view Board. 

Data analysis 

Three team members (SS, DR, and MH) read all 3000 
feedback comments to identify main themes. Themes 
were used to develop a coding sheet to qualify com- 
ments on a goodness/badness, quality/quantity scale as 
well as on the aspect of the curriculum (e.g., questions, 
content or website function) addressed by the com- 
ments. The goodness/badness/quality/quantity scale 
categorized comments as either "bad", "good", "both 
good and bad", "neither good nor bad", "too short/too 
few", "too long/too many", "quantity just right", or "add- 
itional requested features". The curricular aspect/website 
function scale comments were categorized as no mean- 
ingful response, overall quality of the module, factual 
content of module, content organization, module 
images, module length, question quality, question quan- 
tity, or website function. The same three team members 
then re-read each feedback comment and coded them 
on each of the two categorization aspects. Unanimity 
among all three coders was required; when unanimity 
was not present, feedback comments were discussed 
until consensus was reached. 

Average change in scores was calculated by subtracting 
aggregate pretest scores from aggregate post test scores 
for each module. Reliability testing was done on all 
pretests and post tests by performing item discrimination 
on each item and Cronbach's alpha on each test. Content 
of pretests and post tests underwent face validity testing, 
and test results were evaluated by training year to perform 
construct validity. Reliability and validity testing on this 
curriculum has been reported elsewhere [17]. These 
scores were then weighted by the number of learners for 
each module in a given academic year. Then the average 
change in scores was determined among all modules in 
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a given academic year. We used frequency analyses to de- 
scribe the respondent population and comment themes, 
Student's t-test to compare average module ratings, and 
one-way ANOVA to compare average change in module 
score. Tests of significance were two-tailed, with an alpha 
level of 0.05. We performed analyses using SPSS, version 
18.0 (Chicago, IL). 

Results 

Description of learners 

All learners were affiliated with internal medicine resi- 
dency training programs that subscribed to the Johns 
Hopkins Internal Medicine Curriculum through the 
Johns Hopkins Internet Learning Center. Subscribing 
programs included academic medical centers, commu- 
nity hospitals, and military hospitals. In academic year 
2003/04, there were 3670 learners at 50 subscribing pro- 
grams, which grew to 10,235 learners at 122 subscribing 
programs by the 2008/09 academic year. During the aca- 
demic years 2003/04 through 2008/09 a cumulative 
37,755 learners registered with the website. Most lear- 
ners were internal medicine resident or attending physi- 
cians (Table 1), but included a small number of medical 
students, administrative staff, or undefined users 
(grouped as "Other"). Subscribing programs uniformly 
required that resident physicians complete any number 
of the 19 modules studied, while completion by attend- 
ing physicians was voluntary. Learners completed a total 
of 128,850 modules over the six years of study. 

Website capability to gather qualitative feedback from 
learners was added in the 2005/06 academic year, and 
was a mandatory component of module completion. For 
the purposes of this study, a subset of 1000 comments 
was gathered from each of the first three academic years 
that qualitative feedback was available. The training year 
or attending status of learners used in the random sam- 
ple of 3000 feedback comments is shown in Table 1. 

Qualitative analysis 

Thematic analysis of learner feedback identified five 
themes of learner comments: the overall quality of the 



Table 1 Learner demographics 



Year of Training 


All learners* 
N (%) 


Qualitative 
feedback* 
analysis N (%) 


PGY1 


9437 (24.4) 


1019 (34.0) 


PGY2 


9864 (25.5) 


926 (30.9) 


PGY3 


9653 (25.0) 


852 (28.4) 


Attending 


6605 (1 7) 


64 (2.1) 


Other 


3196 (8.2) 


1 39 (4.6) 


Total 


37,755 


3000 



•Academic years 2003/04-2008/09; ^Academic years 2005/06 -2007/08. 



module (mentioned in 32.7 % of comments), the content 
of the module (16.0 % of comments), the questions con- 
tained in the module (4.6 % of comments), the length of 
the module (0.7 % of comments), and website function 
(2.4 % of comments) (Table 2). Further breakdown of 
specific comments is shown in Table 2. Nearly all com- 
ments on the overall quality of the module were positive. 
Of the specific comments on content, almost half were 
concerns about the factual content of the curriculum, 
followed by praise of the factual content. The remainder 
of specific comments on content was on organization 
and images, with the majority of comments requesting 
more summary tables, charts and images. 

Learner comments on module questions most com- 
monly pertained to the number of questions (usually 
asking for more of them) followed by complaints about 
the quality of the questions. For those learners com- 
menting on website function, the majority requested 
additional website features, including a timer function 
and a navigation map for each module. The percentage 
of comments on topics other than the overall quality or 
content of the modules was low. 

Response to feedback 

In response to specific feedback from learners, revisions 
were made annually to curricular content and website 
design/function was enhanced (Figure 1 and Table 3). 
Curricular content was revised to make it more visual, 
with greater numbers of tables, figures, algorithms, and 
images. A small number of demonstration videos were 
added. The number of pretest and post test questions 
was increased and explanatory text for answers on the 
pretest and posttest were added. When appropriate, di- 
dactic sections were summarized with recap statements 
at the end of the section. Website design and function 
was changed to add a navigation map to each module. 
The didactics section was changed so that learners could 
do the didactic sections in the order of their choosing (i. 
e., "learner control"), although they had to complete the 
pretest before the didactics, and could not access the 
post test until after completing all the didactics (i.e., 
"program control"). Finally, factual content of modules 
was updated annually irrespective of feedback comments 
when necessitated by advances in the medical literature. 

Changes In education outcomes 

The average module ratings on the instructive value of 
the 19 modules studied were tracked for the three years 
before and three years after the incorporation of qualita- 
tive feedback in curricular revision (Figure 2). A total of 
56,254 modules were completed in the "before" group, 
with an average instructive value of 3.8. In the "after" 
group, a total of 72,596 modules were completed, with 
an average instructive value of 4.1. The difference 
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Table 2 Qualitative feedback 



Subject 



N (%) 



Examples 



Overall quality of module 

Good 



IVIodule content 

Factual good 

Factual bad 

Length/organization good 
Length/organization bad 
mages/too few 

Module questions 

Quality good 

Quality bad 

Quantity bad 

IVIodule length 

Too long 

Website function 

Good 



Additional features 
requested 

Total 



1026 (32.7) 
1008 (32.2) 

1 8 (0.6) 

503 (16.0) 
154 (4.9) 

219 (7.0) 

55 (1.8) 

32 (1.0) 

43 (1.4) 

144 (4.6) 
1 8 (0.2) 

47 (1.5) 

79 (2.5) 

22 (0.7) 
22 (0.7) 

75 (2.4) 
3 (01) 

1 3 (0.4) 

59 (1.9) 

3135 (100) 



• Awesome module 

• A very good learning module 

• Good 

• Proofread it; there are numerous grammatical errors throughout the module 

• Found this module to be minimally helpful 

• Some mistakes 



• Informative 

• Good review on diabetes treatment 

• Good review of cancer screening 

• Would like to hear more about recent studies 

• Include trade names with generics to make it easier to comprehend 

• The use of bisphosphonates was not entirely clear 

• Liked the diagrams 

• This module was done very well in a stepwise manner 

• Liked the format of the clinical cases as well as the pretest/post test organization 

• More printable tables 

• More charts 

• Summarize "red flags" of back pain somewhere in the module 

• This module could be improved by having pictures 

• More pictures 

• Pictures could have been included (e.g., tonsillar exudates, 
local GABHS complications, scarlet fever rash) 



• Very good set of questions 

• Good questions 

• Very complete in its questions 

• The pretest questions were poorly worded 

• Very poor questions; answers do not correspond with information given 
in the module 

• You can add more clinical settings to the stem of the questions 

• Increase practice questions 

• Ask more questions 

• Less questions 



• Make explanations more concise 

• Too long 

• Was just too lengthy 



• Good organized feedback 

• Easy to use 

• I like that we can download summaries in PDF 

• The case answer was different than the correct answer 

• I wasn't receiving credit for correct questions during the cases 

• The "pop up" had the wrong answer 

• Try to time the modules 

• I think you should give an idea of how many questions are in a section 

• Brief reading material before the start of the module 



between these two scores was statistically significant 
(p < 0.001). 

The average change in scores (i.e., post test minus pre- 
test) was calculated on the 56,254 modules studied in 



the three years before incorporation of qualitative feed- 
back in curricular revision and is compared with the 
average change in scores on the 72,596 modules com- 
pleted after feedback was incorporated in curricular 
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Curricular content revised, 
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Revised content 
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removed; website 
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Juiy 




January 
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Figure 1 Annual repeating cycle of feedback-based curricular revision and website enhancement. A timeline of annual curricular cycle is 
shown. Learners access curricular content throughout the academic year and provide feedback. Curricular content is revised and website 
enhancements are designed annually based on learner feedback. Old content is removed at the end of the academic year and revised content 
and website upgrades are added as the new academic year begins. 



revision (Figure 3). In the "before" group, the average 
improvement in scores was 17.0 %. In the "after" group, 
the average improvement in scores was 20.2 %. In com- 
paring these two groups of changes in scores, these dif- 
ferences were statistically significant (p < 0.001). 

Discussion 

Using qualitative feedback from learners as a needs as- 
sessment to guide curricular revision and website design 
is associated with greater learner satisfaction and larger 
gains in knowledge. Qualitative learner feedback on 
desired improvements centers mostly on the general 
quality of the module rather than any specific compo- 
nent of the content or website function. When making 
specific comments, the greatest number of comments is 
on the educational content of the curriculum, particu- 
larly its factual elements. A small proportion of learners 
request reorganization of content, typically to include 
more tables, charts, figures, and images. Those who 
comment on practice questions typically want more of 
them. A small number of learners comment on website 
function, usually to request additional features. 

While others have commented that taking bad edu- 
cational content and putting it online doesn't improve 
learner satisfaction, we found that responding to 
learner feedback on content that is generally rated as 
good and reorganizing that content (i.e., adding tables, 
figures, and images) and changing website design/ 

Table 3 Summary of curricular and website changes 

Sphere Changes (year added) 

Curricular content 



function in response to specific feedback improved 
learner satisfaction [7]. What we found demonstrates 
the value of recurrent needs assessment in curricular 
revision. Needs assessment should be done when a 
curriculum is first developed, but since learners might 
not fully know their needs until exposed to subject 
matter, and learner needs evolve, recurrent needs as- 
sessment should be built into online curricular design 
[1,2,4,18]. Over time, our curricular content became more 
visual (through the addition of tables, figures, and images) 
and more interactive (through the addition of more ques- 
tions and greater opportunity for feedback). With these 
changes, satisfaction among our learners improved. Get- 
ting learners to repeatedly access an online curriculum 
requires that they be satisfied with it [8,18]. Our website 
design, including a strong evaluation component, and 
our process of using this information as a recurrent 
needs assessment, is one way to increase the chances of 
success of an online curriculum. 

While placing a curriculum online offers additional 
tools to enhance content (i.e., interactivity, hyperlinks, 
audio and video) our results confirm what others have 
shown: the quality of educational content is what mat- 
ters most to learners [7,12,13]. The majority of learner 
comments on the curriculum were related to the overall 
quality of the modules and their educational content, 
suggesting that the educational quality of the content 
was most important to learners. Learner satisfaction is 



• Increased number of tables (all years) 

• Increased number of figures/algorithms (all years) 

• Increased number of images (all years) 

• Increased number of questions on pre/post tests (all years) 

• Added explanatory text on correct/incorrect answers on pretest/post test (2007) 

• Added recap statements at end of didactic sections (all years) 



Website 
design/function 



• Added navigation map to each module (2007) 

• Changed didactic section from program control to learner control (2007)Added video content capability (2005) 

• Added 1-page printable summaries for completed modules (2006) 



Sisson et al. BMC Medical Education 2012, 12:55 
http://www.biomedcentral.com/1472-6920/12/55 



Page 6 of 7 




2003-2006 2006-2009 
Years 



Figure 2 Average instructive value of modules. Average learner 
ratings on instructive value of didactic modules (1 =not instructive; 
5 = highly instructive) are compared between the 2003-2006 
academic years (before qualitative feedback was incorporated into 
annual curricular revision cycle) and the 2006-2009 academic years 
(after qualitative feedback was incorporated into annual curricular 
revision cycle). *Difference is statistically significant (p< 0.001). 



also strongly associated with knowledge gains. In a 
meta-analysis of studies of predictors of learner satisfac- 
tion done before the Internet was widely used, learner 
satisfaction with a curriculum was determined most by 
how much learners felt they got out of the educational 
content (i.e., "teaching effectiveness") [19]. In our study, 
we used the average change in scores on a module (i.e., 
post test score minus pretest score) as a measure of 
teaching effectiveness. We found that by responding to 
learner feedback, the teaching effectiveness of our mod- 
ules improved. Since teaching effectiveness is such a 

/ ^ 



25.0% 
20.0% 
5 15.0% 

01 

c 10.0% 

ra 

.c 

u 





20.2%* 


17.0% 
















2003-2006 


2006-2009 



Years 

Figure 3 Average changes in score per module by academic 
year. Average change in score (post test minus pretest) on didactic 
modules are compared between the 2003-2006 academic years 
(before qualitative feedback was incorporated into annual curricular 
revision cycle) and the 2006-2009 academic years (after qualitative 
feedback was incorporated into annual curricular revision cycle). 
*Difference is statistically significant (p< 0.001). 



driver of learner satisfaction, it is not surprising that our 
results showed similar findings in teaching effectiveness 
and learner satisfaction. Although we are unable to 
determine which changes (i.e., website functionality, 
content organization etc.) contributed to these improve- 
ments, it may not matter. The more important finding is 
that the process of incorporating learner feedback into 
curricular revision is associated with improved outcomes 
in learner satisfaction and knowledge outcomes. 

It is our opinion that online curricula should be 
designed to include powerful evaluation tools (including 
learner satisfaction as described here) to assess the reli- 
ability and validity of education outcomes. We have con- 
tinually leveraged the capabilities of being online to 
increase the evaluation component of our curriculum. 
Assessment tools (i.e., pretests and post tests) were 
expanded, and item discrimination and Cronbach's alpha 
calculations were added to these instruments. Group 
performance measures, subgroup (i.e., training year; 
training program) performance measures, and individ- 
ual performance measures (including standard scores) 
were added or expanded. This evaluation component 
allowed us not only to perform a continuous needs as- 
sessment of our learners, but also to serve the needs 
of the residency program directors that chose to im- 
plement our curriculum. The Accreditation Council of 
Graduate Medical Education (ACGME) requires that 
program directors demonstrate the educational out- 
comes of their curricula, which is provided at a group 
and individual level by our curriculum [20]. Use of our 
curriculum has grown from 24 internal medicine resi- 
dency programs in 2002 to over 160 programs in 
2011. As others have pointed out, tracking evaluation 
outcomes and learner feedback consumes resources, as 
does the resultant curricular revision and implementa- 
tion of website improvements, but these costs are 
minimized by sharing resources among users [4,5]. 
Developing a strong evaluation component to online 
curricula allows educators to advance the science of 
education and answer the call to increase evaluative 
research on education outcomes and those features 
that improve them [15,21,22]. 

Our study has several limitations. Individual learners 
changed from year to year, and so changes in learner sat- 
isfaction and knowledge outcomes may have been due 
to learner characteristics (e.g., post-graduate year, pro- 
gram type) rather than the curriculum. Since several 
changes to curricular content and website design were 
made in any given year, we were unable to measure the 
impact of any single change on learner satisfaction or 
knowledge outcomes. Over the years of study, online 
education resources expanded greatly, and learners may 
in general have become more satisfied with online didac- 
tics independent of our changes. Also over the years of 
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Study, clinician educators who wrote the educational 
content may have improved their writing skills, contrib- 
uting to improvement in the educational content of the 
curriculum. 

Conclusions 

We found that when asked to give feedback on how to 
improve an online curriculum, learners commented 
most on its educational content rather than its inter- 
activity or website design. We also found that a process 
of recurrent needs assessment followed by curricular re- 
vision is associated with improved learner satisfaction 
and larger gains in knowledge. Online educational cur- 
ricula should be designed to include evaluative tools that 
measure educational outcomes for study so that educa- 
tional outcomes can be improved. 
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